Papers with Vector Quantization
SCVQ: Sparse-Compensated Vector Quantization for Large Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Existing vector quantization methods incur inference overhead due to massive codebook storage and intensive index lookups. |
| Approach: | They propose a framework for vector quantization that incorporates a salience-aware weighted K-means clustering scheme with symmetry constraints to reduce codebook size and indexing costs. |
| Outcome: | The proposed framework achieves a perplexity of 5.78 on WikiText-2 for LLaMA-2-7B at 2-bit quantization while delivering a 1.4 speedup over existing baselines. |
VPTQ: Extreme Low-bit Vector Post-Training Quantization for Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent research has focused on pushing weight-only quantization to extremely low-bit due to numerical representation limitations. |
| Approach: | They propose a vector-based quantization approach that pushes LLMs to extremely low-bit . they propose scalar-based weight quantization that reduces memory requirements and optimizes storage costs . |
| Outcome: | The proposed method reduces model quantization perplexity by 0.01-0.34 on LLaMA-2, 0.38-0.68 on mistral-7B, 4.41-7.34, on llaMA-3 on QA tasks on average. |